1 10 443 canary-openshift-ingress-canary.apps.sno.eformat.me
1 10 443 canary-openshift-ingress-canary.apps.baz.eformat.me
Tag: service discovery
Service Discovery and Load Balancing with Stork
25 May 2023
Tags : quarkus, service discovery, stork, load balancing, java
Stork is a service discovery and client-side load-balancing framework. Its one of those critical services you find out you need when doing distributed services programming. Have a read of the docs, it integrates into common open source tooling such as Hashi’s Consul as well as a host of others. Even-though OpenShift/Kubernetes has a built-in support for service discovery and load-balancing, you may need more flexibility to carefully select the service instance you want.
DNS SRV for Service Discovery
I wanted to try out good 'ol fashioned SRV Records as a means to testing out the client side service discovery in Stork. Many people forget that DNS itself supports service discovery for high service availability. It is still very commonly used, especially in mobile/telco.
My test case would be to create a DNS SRV record that queries OpenShift Cluster Canary Application endpoints. I’m using Route53 for DNS so you can read the SRV Records format here. The first three records are priority, weight, and port.
If you curl one of these, you get a Healthcheck requested back if the service is running.
curl https://canary-openshift-ingress-canary.apps.sno.eformat.me
Healthcheck requestedSo, in my example, you can get a full list of SRV record values by querying:
dig SRV canary.demo.redhatlabs.devCoding a Quick Client
For a quick and dirty client to make use of the SRV record I reach out for my favourite tools, yes Perl 🐫🐫🐫 !
Let’s query the SRV record and see if my OpenShift clusters are healthy.
# sudo dnf install -y perl-Net-DNS perl-WWW-Curl
use Net::DNS;
use WWW::Curl::Easy;
use Term::ANSIColor qw(:constants);
sub lookup {
my ($dc) = @_;
my $res = Net::DNS::Resolver-> new;
my $query = $res->send($dc, "SRV");
if ($query) {
foreach $rr ($query->answer) {
next unless $rr->type eq 'SRV';
# return first found
return $rr->target;
}
} else {
print("SRV lookup failed: " . $res->errorstring);
}
return;
}
my $host = lookup("canary.demo.redhatlabs.dev");
print GREEN, $host . "\n", RESET;
my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER,1);
$curl->setopt(CURLOPT_URL, 'https://' . $host);
$curl->setopt(CURLOPT_SSL_VERIFYHOST, 0);
my $retcode = $curl->perform;
if ($retcode == 0) {
print("Transfer went ok\n");
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE);
print(GREEN, "Received response code: $response_code\n", RESET, "\n");
} else {
print(RED, "An error happened: $retcode ". RESET . $curl->strerror($retcode)." ".$curl->errbuf."\n");
}Of course feel free to run this in a loop :) because each record is equally weighted in the SRV you will get a round-robin behaviour.
So, looking good so far.
Stork and Java
Of course, the whole point was to try out Stork. Following the Quarkus Stork getting started guide, I used a simple rest client service
and configured the canary stork service as follows:
Unfortunately, this didn’t work as I expected! The SRV record values were resolved to IP addresses instead of returning the DNS name for me to query. The
issue with just an IP address is that Routing in OpenShift requires the HEAD/Location to be set properly so the correct endpoint Route can be routed and queried using HAProxy.
The Stork documentation spells out this DNS resolution process:
Looking at the source code, led me to submit this PR which adds in an option so that you can skip the DNS resolution step.
So, adding this property using the new version of the Stork library:
quarkus.stork.canary.service-discovery.resolve-srv=falseLeads to the DNS names being returned and no the ip addresses:
Trying out the code and now the Service call works as expected:
YAY ! 🦍 Checkout the source code here and watch out for the next version of Stork !
